X-OriginalArrivalTime: 18 Nov 2004 00:44:05.0759 (UTC) FILETIME=[B4F448F0:01C4CD07] 
Subject: NIED Problem 
Date: Wed, 17 Nov 2004 18:44:01 -0600 
From: "Imafuku, Nobuhiro" <Nobuhiro.Imafuku@mts.com> 
To: "Rood, Omar" <Omar.Rood@mts.com> 
Cc: "Krantz, Don" <Don.Krantz@mts.com>, 
        "Larson, Rod" <Rod.Larson@mts.com>, 
        "Nelson, Terry" <Terry.Nelson@mts.com>, 
        "Thoen, Brad" <Brad.Thoen@mts.com>, 
        "Sparks, Hugh" <hugh@csparks.com>, 
        "Shigemasa, Yukio" <Yukio.Shigemasa@mts.com>, 
        "Nakashima, Toshiaki" <Toshiaki.Nakashima@mts.com>, 
        "Otsuka, Yutaka" <Yutaka.Otsuka@mts.com> 

Hello, OMAR san.

We had a serious controller communication problem 16th November. There
are no oil leakage and mechanical damage. Follows are summery of
official letter to NIED written by MHI.

     ====================

     1. Outbreak Day

          16-November-2004

     2. Phenomenon

Table controller send alarm signal to MHI operation controller
performing hydraulic sequence control due to happen communication
trouble between Y servo valve controller and Z servo valve controller
during static displacement input/output error test. Then, MHI operation
controller perform level E emergency process (stop applying hydraulic
pressure by closing emergency shut-off valve). Table is normal position
(X = Y = Z = 0 mm) when communication error happen. Table move slowly to
table stand direction by table own weight due to shutting down
hydraulic, then, table stop Z actuator displacement end position (X = 0
mm, Y = 400 mm, Z = - 540 mm). It happen big sound during performing
emergency sequence, but table does not go unstable.

     3. Cause

          MTS is trying to investigate cause of happening communication
error between Y servo valve controller and Z servo valve controller.

     ====================

Current another problem is MHI stop to apply hydraulic pressure until
finding cause of this problem clearly. We (with ROD & TERRY) are trying
to find cause now. But, we do not have any good idea to check this
problem. Please respond any of your idea to check this problem.

REGARDS     FUKU



Subject: NIED 
Date: Tue, 16 Nov 2004 07:36:13 -0600 
From: "Rod Larson" <rod.larson@mts.com> 
To: "Thoen, Brad" <brad.thoen@mts.com>, 
        "Sparks, Hugh" <hugh.sparks@mts.com> 

Brad/Hugh,


Just arrived here in Seishin-Chuo and talked to Terry Nelson.
The crashes on the scram-net are I think 2 distinct different problems. The
one that happened over the weekend is the classic power-off of one
console...Since over the weekend...we do have to wonder about the
possiblility of the darn cleaning lady again. She is notorious for
unplugging stuff to plug in her vacuum....including PCs...


Now the one that Terry reported that happened yesterday about 5pm is
different...There the Grand Connector turns white (just white box) then
several units crash...Y/Z S/V Controllers lost Scram-net communication and
the Grand Connector errored...


I think this 2nd error occurs when Office 2000 is running. MHI was using
Excel to save data from panels during static tests...I have seen this
before...Office 2000 seems to kill the Scram-net once in a while..It is NOT
predictable as to when it crashes...


One more item..it seems that Yaw is messed up..i have to check it but it
does not seem to have the right proportion of error going to the
axes....the table is not square so you cannot drive the X actuators like
the Y...or...maybe it is because of # of servovalves on each axis...that
could hurt tooo..

Rod


From: "Hugh Sparks" <hugh@csparks.com> 
To: "Rod Larson" <rod.larson@mts.com> 
Cc: "Brad Thoen" <brad.thoen@mts.com> 
Subject: Re: NIED 
Date: Tue, 16 Nov 2004 08:26:23 -0600 


>Now the one that Terry reported that happened yesterday about 5pm is
>different...There the Grand Connector turns white (just white box) then
>several units crash...Y/Z S/V Controllers lost Scram-net communication and
>the Grand Connector errored...


>Office 2000 seems to kill the Scram-net once in a while..It is NOT
>predictable as to when it crashes...


Office is not killing the scramnet. It is causing the network connection
monitor that is part of the Grand Connector  and Table Controller
hosts to decide that the valve controller host has gone away. This
timeout  is adjustable, but the longer we make this time, the longer
will take to notice a real failure.


Microsoft Office and many other applications block the execution
of other Windows programs whenever they feel the need. Many
operations in windows will cause this problem:


Mounting a network drive from a Windows host will sometime block
the running applications for 20 seconds or more. (Sometimes forever:
the whole desktop crashes when setting up file shares now and then.)
Even using the GUI components of other applications under Windows
can stop other programs from runing indefinitely. The file-picker in
Office is a notorious example.


Alltalk goes to great lengths to avoid doing the kinds of things
that block real-time repsonse. We can't offer any guarantees
if the customer insists on running other applications. Raging
and screaming will not alter this fact in the least degree.


I'm certain that we will never reliably support running things
like video games or Microsoft Office on our controller hosts
while they are talking to the VME box. Our hosts need to
respond to data aquisition events, file playback buffer requests,
as well as networking events when we have a distributed system.


I have consistently warned everyone about this for many many
years. These warnings have generally been ignored.
These are controllers. Not play-toys.


Our controller GUI programs must respond to real time events on the
network fairly quickly. If we turn this feature off, we will not be able
to respond to real emergencies.


I believe that it may be possible to provide for background-priority
file transfers to other, non-realtime machines where they can run
Office, Matlab,  or whatever they like. We would obviously have to
spend some time and money to investigate this possibility. I believe this
is the best solution.


Another possibility is to eliminate the ability to start and stop
individual valve controllers and have them automatically reconnected.
The system would have to shut down and started up as a whole.
If we eliminate this feature AND get rid of the Grand Connector
monitoring for live connections, there would be no need for network
traffic between the hosts after the system has been started and
interconnected. I don't think this is a very good idea.


Another method, which would require considerable effort, would
be to run the supervisor application at an elevated priority on
each host. This would also require moving the distributed network
support, data acquistion and drive playback features out of Alltalk
and putting them in the supervisor.


This is not a guaranteed fix because I don't know if the Windows
socket support API is thread-safe. It also introduces a world of
difficulties with deadlocks, timeouts and other frustrations that
plague our friends downstairs.


Our GUI hosts at NIED must be treated like dedicated controllers,
not general-purpose PCs. This is particularly vital when a real-time
network is involved.


-Hugh Sparks, hugh@csparks.com


From: "Hugh Sparks" <hugh@csparks.com> 
To: "Larson, Rod" <Rod.Larson@mts.com>, "Nelson, Terry" <Terry.Nelson@mts.com>, 
        "Brad Thoen" <brad.thoen@mts.com> 
Cc: "Rood, Omar" <Omar.Rood@mts.com> 
Subject: Timeout adjustments at NIED 
Date: Thu, 18 Nov 2004 02:55:22 -0600 


Before you get happy changing timeouts, consider these
ideas:

The default timeout values took weeks to devise and test
and they can interact with the hosts in complex ways I
can't go into tonight. If you make big changes, something
else will almost certainly break. Be conservative.

It is not proven that the network "failed" at all.
It may be simply doing its job and telling you that
a DSP crashed.

Any sort of hardware glitch that causes a floating point
or memory access exception will blow up a DSP and give
very similar symptoms.

None of the timeouts are likely to occur when "nothing is going on",
so the weekend crash you mentioned earlier must be due to some other
software failure: Whatever happened on the weekend when "nothing
was going on" is <probably> what happened when the table was running
on Nov. 16. 

A power-save feature or screen saver could cause problems, but
I don't think that's very likely because we have run night and
day for weeks at a time. Still, someone might have turned one
on, so you need to check all the hosts for tampering.

IMHO:

If we promised NIED that they could play with Microsoft Office
on the controller hosts, we should take our beating now and
retract this claim. Much worse will follow if we don't prevent
these activities. I honestly think it would be best to uninstall
Office and all other non-essentail programs on the host machines.
 
If we allow the customers to install arbitrary applications and
run them on the valve or table controller hosts, we are absolutely
certain to have problems. Someone in a position of authority
at NIED and/or Mitsubishi has to intervene and control this
behavior or serious damage is almost a certainty.

End rant.

*************************

Each node has it's own set of timeout variables. By default
they are all identical. Some hosts alter the read timeout
value when they know that a special message will take
a very long time. This is done, for example, when a host
creates the virtual machine on the dsp at startup.

To alter the timing variables, first open a listener window
on the selected host.

Execute this expression:

    WITH Link

This gets you into a context where the timeout variables are
visible.

Now you can change timeout values using assignment statements.
The effect will be immediate for the read, write and ping
related timouts. The connect and async timeouts are only used
when connections are created. They are not likely to require
changes.

Don't forget the semi-colon when using the listener.
Shown below are the variables and the default values.
The times are in 1/100 second tics.

 readTimeout     := 1000 ;  ! Read timeout
 writeTimeout    := 1000 ;  ! Write timeout
 connectTimeout  := 5 ;     ! Connect timeout
 asyncTimeout    := 100 ;   ! Asych connect timeout
 pingTime        := 25 ;    ! Time between sending pings
 pingTimeout     := 500 ;   ! Link idle timeout

The only timeouts you will be likely to fuss with are
the pingTime and pingTimeout which are explained below.

To understand these variables, you need a little overview
of how connections operate:

Connections between nodes are directed: there is a source
node and a destination node. The source node can send synchronous
messages to the destination node. A synchronous message is a 
message with parameters that returns a value to the sender.
The sender is blocked until the value is received. For example,
if x is a variable that contains a remote object (on some other
node) that understands the #size message, you could execute
the expression:

    y := x.size + 3

The message #size is sent to the remote object stored in x.
The value that gets returned is then added to 3 locally and
the result is stored in y. The sender gets blocked while
the message is in transit, during the execution on the remote
node, and during the time required to return the value.

An asynchronous message is sent from the destination node back to
the source. Asynchronous messages are used for things like buffer
requests during data acquisition or drive file playback. They are
also used for certain kinds of error that are not directly caused
by an incoming message. The sender of an asynchronous message doesn't
wait for a reply. This is important when a periodic real-time
method needs to send a message up to the host.

A connection consists of two channels. A synchronous channel
and an asynchronous channel. Each channel has its own socket
pair on the sending and receiving side.

The asynchronous channel is used for the asynchronous messages
described above. It is also used to send keep-alive messages
called "pings" from the destination node back to the source node.

The destination node sends pings up the asynchronous channel
at a rate controlled by the pingTime variable. This time is a
minimum value: if the host is very busy, it won't send as many
pings.
 
The source node decides that the destination node is "gone" if
no pings arrive for more than a certain time limit.
This time is the "pingTimeout" interval listed above.

That's why the pingTime is much shorter than the pingTimeout.
It only takes on ping within the pingTimeout interval to satisfy a
host that the destination is still alive.
 
If a host is "bothering" a DSP with lots of messages so it
doesn't have time to send back pings, that's ok because
any completed synchronous message is also considered a
successful ping.

A destination node has to hang up for the pingTimeout, sending
no pings AND not process any synchronous messages before the
source node decides that it has gone away.

When a host requests a connection to a remote node, it can
go on doing other things while the connection is being created.
The host can configure a notification message that will be
sent when the connection is completed and also when the connection
goes down. A parameter sent with the message tells the receiver
if the connection is "up" or "down".

Some connections are configured to send an error notifications
if the connection goes down. This is how the Grand Connector
decides that a host has gone away. The most likely place you
will want to fuss with pingTime and pingTimeout is on the
Grand Connector. For example, if the Table Controller got
very busy doing lots of stuff so that it would not respond
to the GC for more than 5 seconds, that would be Bad Thing.

It is also possible for a valve controller DSP to get tied up
by lots of messages from the Table Controller so that its own
host will get mad and decide that the connection has
gone down.

The read and write timeouts are 10 seconds by default.
That is a long time for a computer to "go away".
If a read or write timeout occurs, it is a serious error.
The read and write operations are used to transfer messages
that would be corrupted if incomplete.

The connectTimeout is short because the Grand Connector
uses this to find out if another host is ready. It is not an
error if the remote host is not ready. When the connect succeeds,
the GC goes ahead and hooks up all the wires.

The asyncTimeout is used when forming reverse connections
for upward messages. It is only used immediately after making
a successful synchronous connection. It is very unlikely that
you will ever need to change this value.


*****************

The ten second hangup on the Grand Connector was the
10 second read or write timeout that decides that
a serious error has occurred. The GC doesn't try to
talk directly with a DSP after the system is configured.
It only talks to the other hosts to see if they are
still alive. The message about node 14 (table DSP) being
gone probably came from the Table Controller, not the
Grand Connector.

To sort out the network vs DSP problem, I need to see more
information from the Hyperterminal and listener windows on
all the hosts.

I'll talk to you again tomorrow.

-Hugh

Mail: hugh.sparks@mts.com
Office: (952) 955-2800
Cell: (612) 247-2714
Home: (952) 955-2800
